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ABSTRACT 

Objective: To establish the role of high-fidelity 
simulation training to test the efficacy and safety of the 
electronic health record (EHR)-user interface within the 
intensive care unit (ICU) environment. 
Design: Prospective pilot study. 
Setting: Medical ICU in an academic medical centre. 
Participants: Postgraduate medical trainees. 
Interventions: A 5-day-simulated ICU patient was 
developed in the EHR including labs, hourly vitals, 
medication administration, ventilator settings, nursing 
and notes. Fourteen medical issues requiring 
recognition and subsequent changes in management 
were included. Issues were chosen based on their 
frequency of occurrence within the ICU and their ability 
to test different aspects of the EHR-user interface. ICU 
residents, blinded to the presence of medical errors 
within the case, were provided a sign-out and given 
10 min to review the case in the EHR. They then 
presented the case with their management suggestions 
to an attending physician. Participants were graded on 
the number of issues identified. All participants were 
provided with immediate feedback upon completion of 
the simulation. 

Primary and secondary outcomes: To determine 
the frequency of error recognition in an EHR 
simulation. To determine factors associated with 
improved performance in the simulation. 
Results: 38 participants including 9 interns, 10 
residents and 19 fellows were tested. The average error 
recognition rate was 41% (range 6-73%), which 
increased slightly with the level of training (35%, 41% 
and 50% for interns, residents, and fellows, 
respectively). Over-sedation was the least-recognised 
error (16%); poor glycemic control was most often 
recognised (68%). Only 32% of the participants 
recognised inappropriate antibiotic dosing. 
Performance correlated with the total number of 
screens used (p=0.03). 

Conclusions: Despite development of comprehensive 
EHRs, there remain significant gaps in identifying 
dangerous medical management issues. This gap 
remains despite high levels of medical training, 
suggesting that EHR-specific training may be 
beneficial. Simulation provides a novel tool in order to 
both identify these gaps as well as foster EHR-specific 
training. 



ARTICLE SUMMARY 



Article focus 

■ Developing a simulation environment to test 
ability of providers to recognise medical errors in 
the EHR. 

■ To establish the reproducibility of EHR-based 
simulation testing. 

■ To understand the types of medical errors/ 
patient trends which are not recognised by the 
average user of the EHR. 

Key messages 

■ Average users of the EHR, irrespective of the 
level of training, have a poor rate of recognising 
disturbing trends in patient condition or medical 
errors. 

■ Simulation testing will allow for a structured way 
to both restructure EHR education as well as 
redesign. 

■ Issues related to the EHR-user interface are 
magnified by the data-rich intensive care unit 
environment. 

Strengths and limitations of this study 

■ The study demonstrates the feasibility of using 
EHR simulation to identify patient safety and 
quality issues related to the EHR/user interface. 

■ The study provides a framework to test how new 
educational techniques or EHR interface design 
can improve patient safety and error recognition. 

■ This pilot study does not address whether par- 
ticipation in the simulation itself improves the 
provider use of the EHR. 



INTRODUCTION 

Use of the electronic health record (EHR) is 
growing in the USA, spurred by financial 
incentives from the American Recovery and 
Reinvestment Act (ARRA). 1 2 A growing 
body of research demonstrates that EHRs 
provide a myriad of benefits, including 
increased adherence to guideline-based care, 
decreased prescribing errors and improved 
disease monitoring. 3-5 There has been a sig- 
nificant rise in EHR use across the country, 
with a near tripling in the number of 
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hospitals using any form of EHR during the first decade 
of the 21st century. 6 7 By the end of 2011, EHR adoption 
had increased to over 50% of all US physicians, stimu- 
lated by $2.5 billion in incentives paid out under the 
Health Information Technology for Clinical and 
Economic Health (HITECH) Act of ARRA. 8 As even 
more healthcare systems transition to EHRs, there will 
be an increasing need for the development of new 
methods to effectively train healthcare providers, par- 
ticularly with respect to maximising the functionality of 
the EHR as a clinical tool. 

While EHRs can offer significant benefits, they can 
also foster errors in ways that paper documentation did 
not, a phenomenon that has been termed 
'e-iatrogenesis'. At the most fundamental level, EHR 
software itself can be poorly designed and may promote 
errors such as radiation overdosing or miscalculating 
patient medication doses. 10 Medication ordering and 
monitoring appear to be particularly vulnerable to 
errors in the EHR. Duplicate medication orders, as well 
as drug dosing and monitoring errors have been shown 
to increase in the post-EHR era. 11 12 More complex 
types of errors arise from the way clinicians interface 
with the EHR; many of these errors were unforeseen 
prior to implementation of these systems. 13 The com- 
plexity of EHR implementations has often led to unin- 
tended consequences and errors and recent studies have 
evaluated the concept of fragmentation of the 'big 
picture' of a patient's trajectory by the vast amount of 
information displayed in a patient's electronic record 
and the resultant data overload inflicted on the clini- 

14 15 

cian s cognitive process. 

In November 2011, the Institute of Medicine (IOM) 
released a report on the safety of health information 
technology (HIT) 16 that detailed challenges associated 
with the safe implementation of HIT. The report docu- 
mented both the predictable and the unintended conse- 
quences of EHRs. This report also developed a 
taxonomy for classification of errors with categories that 
included data fragmentation, over-completeness (includ- 
ing excessive redundancy and copy-and-paste), errors in 
data recognition and perhaps most importandy cogni- 
tive errors. 1 *' 17 The latter arise when users are unable to 
effectively process data to make appropriate decisions 
due to the method by which data are presented within 
the EHR. 18 

These safety issues are perhaps most relevant in the 
intensive care unit (ICU) , where a 24 h cycle typically 
generates over 1300 new data points in the health 
record for an average patient. 19 Many of the reports of 
increased errors, patient morbidity and the failure to 
successfully implement EHRs have come from the ICU 
environment. 20 21 In an attempt to address this problem, 
Ahmed et al described a new EHR interface for their 
ICU designed to present data in a context-specific and 
streamlined manner. It was successful at reducing both 
the total amount of errors per provider and the 'task- 
load' index, an indirect measure of data overload. 22 



Unfortunately, most institutions do not have the expert- 
ise or resources to design their own EHR interface, 
instead relying on commercial systems. 

Adequately training providers is a key component 
which may improve EHR safety. Studies document that 
physician training in EHR use is currendy suboptimal. 
Underwood et al demonstrated that while at least 3- 
5 days of training was required for physicians to report 
the highest levels of satisfaction, nearly half the physi- 
cians studied (49.3%) revealed that they had received 
three or fewer days of training. Interestingly, respondent 
ratings on the ease of use for meaningful use measures 
continued to improve with more than 2 weeks of train- 
ing. 23 The IOM and the American Medical Informatics 
Association (AMIA) have identified EHR development, 
implementation and training as key areas for new 
research to improve healthcare quality and safety. 10 24 

In spite of the growth of medical simulation and the 
increasing emphasis on high-fidelity simulation, litde has 
been performed with EHR-specific simulation training. 
Simulation training is particularly attractive as it conveys 
no risk to patients, maintains patient privacy and allows 
a highly specific and reproducible training environment 
that can be tailored to the needs of learners and health- 
care organisations. 25 In order for full task training via 
simulation to be effective, however, there must be spe- 
cific attention given to creating psychological and func- 
tional fidelity, that is, recreating the true 'feel' of the 
goal environment. 26 The few studies on EHR simulation 
have not been in the ICU nor have they truly tested 
physician ability to recognise and process information 
(as opposed to order entry). 27 28 Barnato et alwere suc- 
cessful in creating a realistic simulated ICU environment 
to test decision-making variability in patient triage. 29 
However, in their study, the EHR was utilised as a tool 
within the simulation as opposed to the focus of the 
simulation exercise itself. 

The goal of our study was to create a highly realistic 
and complex-simulated ICU patient encounter in the 
EHR. We developed this simulation as a pilot as part of 
a longer term goal to teach effective use of the EHR in 
the ICU to identify common EHR error types such as 
medication monitoring errors and failure to identify 
concerning trends in laboratory or vital statistics data, 
and to help physicians cope with data fragmentation/ 
overload. 



METHODS 
Ethics statement 

The study was approved by the Oregon Health and 
Science University Institutional Review Board (IRB). 
The study was deemed minimal risk and informed 
consent was not required. All participants were provided 
an IRB-approved information sheet about the protocol. 
All data were de-identified and stored in a secure file. 
The authors are willing to share any and all data 
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obtained from this research. They will be available via 
email to the corresponding author. 

For the study, a new training environment was created 
within our enterprise-wide EHR (EPIC Care; Epic 
Systems, Madison, Wisconsin, USA) that allowed the gen- 
eration of patient cases with multiple consecutive days of 
patient data. This was in contrast to the previous training 
environment that supported only single-day encounters 
as all data were deleted at the end of each day. The new 
training environment was an exact replica of the physi- 
cian's current practice environment; any user-specific 
settings and customisations generated in actual patient 
care were retained in the simulation environment (eg, 
individual preference lists, screen view settings, etc) . 

Within this new environment, we created a multiday- 
simulated Medical ICU (MICU) patient case, which 
detailed the clinical course of a 74-year-old patient with 
diabetics admitted in septic shock with resulting acute 
renal failure and acute respiratory distress syndrome 
(ARDS) requiring mechanical ventilation. The patient 
improved clinically over the initial 48 h, including reso- 
lution of renal failure, shock and fever. Recurrent sepsis 
developed on the fifth hospital day, presumably due to 
an inadequate antibiotic dose in the setting of normal- 
isation of renal function. The case was made as robust as 
possible and included hourly vital signs, a full medica- 
tion administration report (MAR) including as-needed 
(PRN) medications, a detailed hourly intake/output 
report, and nursing, resident, attending and respiratory 
therapy notes. 

The case was designed with the central theme of 
determining whether a diagnosis of recurrent sepsis 
would be made. We chose sepsis as the focus because of 



its high prevalence (it is the leading cause of death in 
the ICU) , the fact that a significant percentage of physi- 
cians believe that this diagnosis is missed in patients and 
epidemiological studies that suggest many patients 
experience a delay in diagnosis which is associated with 
worse outcomes. 30 31 Aside from the physiological and 
laboratory data associated with the diagnosis, we built in 
additional errors which we identified after integrating 
discussion of EHR use into our weekly MICU Morbidity 
and Mortality conference as occurring at a high fre- 
quency. The total number of errors/ patient trends 
within the case was typical for patients with significant 
missed clinical deterioration, particularly in those cases 
where clinical decision-making did not meet best prac- 
tices. In total, 14 individual action items were built in 
the case that could be grouped into the following three 
categories: (1) dangerous trends in lab results or vital 
signs (eg, 25% reduction in blood pressure with tachy- 
cardia and leukocytosis), (2) clear medication errors 
(eg, incorrect antibiotic dose for renal function) and 
(3) failure to adhere to institutional or national best 
practices across critical care (eg, attention to items that 
are covered by the 'FAST HUG' (Feeding, Analgesia, 
Sedation, Thromboembolic prevention, Ulcer prophy- 
laxis, Head of Bed elevation and Glycemic Control)). 32 
Table 1 presents a complete list with definitions of the 
errors included in the case as well as the type of error 
occurring at the EHR-user interface representing each 
specific item and specifically in relation to our institu- 
tion's specific EHR. 

The simulated case was then deployed on an EHR 
workstation in the MICU. Participants included interns, 
residents ( predominandy internal medicine trainees) , as 



Table 1 Fourteen errors developed throughout the 5-day ICU course 


Error safety issue 


EHR category 


Changes in patient condition 




25% Drop in mean arterial pressure, 25% increase in heart rate 


Structure and time, cognition and customisation 


Recurrent sepsis 


Cognition 


Increasing plateau pressure to >30 


Overcompleteness, data finding 


Increase in WBC* 


Structure and time, cognition and customisation 


New fever 


Structure and time, cognition and customisation 


Medication errors 




Inappropriate antibiotic dose (2) 


Data finding, cognition 


Low antibiotic trough 


Data finding, cognition 


Use of D5W in hyperglycemic patient 


Data finding and overcompleteness 


Failure to adhere to best practice 




Glucose>200 mg/dl 


Overcompleteness and data finding 


Tidal volume of 8 cc/kg IBW in acute respiratory distress syndrome 


Data finding and cognition 


Over-sedation 


Data finding 


Lack of daily awakenings 


Data finding 


Recognition of fluid balancet 


Data finding 


They include improper medication dosing or administration, failure to adhere to ICU best practices and inability to identify dangerous patient 


trends. EHR categories are defined as in Ash era/. 




*Net 30% increase in WBC from days 3 to 5. 




tNet 16 litres positive since admission. 




D5W, EHR, electronic health record; IBW, ideal body weight; ICU, intensive care unit; WBC, white blood cell count. 



March CA, Steiger D, Scholl G, era/. BMJ Open 201 3;3:e002549. doi:1 0.1 136/bmjopen-201 3-002549 



3 



Electronic health record simulation 



well as pulmonary, medical and anesthesia critical care 
fellows (of all years of training). All participants had 
received institution-specific training with our EHR and 
had already been users of the system prior to testing. 
This training was standard for all residents and fellows at 
the beginning of their training and comprised of 
1.5 days of small group instruction with one of the 
institution-dedicated EHR trainers. Training involved 
hands on use with the system and included tasks such as 
data retrieval, data entry and instructions on customisa- 
tion. Users were expected to complete a set number of 
tasks in each of these areas prior to completion. Each 
participant was provided a one-page description of the 
patient, including a brief synopsis of the history and a 
current physical examination for the context. 
Participants were told to analyse patient data in order to 
prepare to 'sign-out' the patient to a colleague, includ- 
ing any management changes they would recommend 
making to the patient's care. Participants were blinded 
to the presence of known errors built in the case. Each 
participant used their own login credentials, which 
allowed their own personal EHR customisations to be 
activated within the EHR, and was allotted lOmin of 
chart review time which represents the approximate 
amount of time the average resident spends reviewing 
the chart while prerounding on an individual patient at 
our institution. Of note, we initially tested the case with 
two senior critical care fellows to ensure both its realism 
(in terms of data presentation) and feasibility of comple- 
tion in the allotted time. 

During the exercise, participants were directly 
observed by a member of the study team and all data 
recorded on a standardised data collection sheet. The 
observer noted both the absolute number of screens 
used in reviewing the patient record as well as the use of 
either of two 'high-yield' screens. One of these screens 
(the 'MD Index' screen) was a gateway into multiple dif- 
ferent modes of data presentation, while the other (the 
'Synopsis' screen) presented a graphical view of vital 
sign trends alongside timed MAR and lab data. Of note, 
while all of our primary data and portal screens were 
designed to be used within the ICU environment, none 
are specific to the ICU and they are utilised throughout 
the inpatient environment. 

Each participant made a brief presentation to a member 
of the study team with specific focus on action items that 
should be addressed. The presentation was structured to 
mimic the workflow on daily rounds. Participants were 
scored based on whether they identified the action items/ 
clinical trends within the case. Upon the conclusion of the 
encounter, all participants were given immediate feedback 
on which issues were correct, which were missed, and 
where to find the missing data in the EHR. 

Differences between groups were analysed using a two- 
tailed student t test. Correlations were analysed via 
Spearman's test. (For both, a p value <0.05 was consid- 
ered significant.) All data were analysed with GraphPad 
Prism (San Diego, California, USA). 
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Figure 1 Simulation performance is loosely correlated with 
the level of training. Thirty-nine participants underwent EHR 
simulation and graded according to the number of correctly 
identified errors. Data analysed by analysis of variance. 

RESULTS 

A total of 38 participants were tested: 19 fellows, 10 resi- 
dents and 9 interns. Of the 14 possible medical issues 
requiring recognition and alteration in management, an 
average of 41% (range 6-73%) were identified (figure 1). 
The recognition rate increased significantly with the level 
of clinical training: interns, residents and fellows recog- 
nised 35%, 41% and 50%, respectively (p=0.03; figure 1). 

Overall, there was little consistency in the type of 
errors missed across the cohort as a whole. The least 
recognised issues were the over-sedation of the patient, 
and the lack of daily awakenings (16%), the latter of 
which was indicated by a Motor Activity Assessment 
Scale (MAAS) score varying between zero (unresponsive 
to noxious stimuli) and one (responsive only to noxious 
stimuli). 33 Poor glycemic control was identified but at a 
relatively low rate (68%; figure 2). Of greater concern, 
only 29% correctly recognised the change in vital signs 
consistent with recurrent sepsis. 

Of note, during the first round of testing, we inadvert- 
ently introduced an additional error into the laboratory 
screen when we built the simulated case. The patient, 
instead of having 20% band forms in their manual dif- 
ferential, had 20% basophils. Only 1 of the 14 people 
noted this abnormality, providing additional evidence 
for the potential for the simulation to assess juxtapos- 
ition errors as well the extent to which they exist. Finally, 
except for recognition of an excessive tidal volume 
(>6 cc/kg) (58% vs 21%; p=0.045) and lack of daily awa- 
kenings (53% vs 16%; p=0.038), two best practices for 
intubated patients with ARDS, 34 3 there were no statis- 
tical differences between fellows and residents in recog- 
nition of other errors or safety issues ( figure 3) . Overall, 
the average participant visited 16.4 different screens (an 
average of 35.6 s per screen). The number of individual 
screens visited correlated with the number of errors 
recognised (figure 4). 
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Figure 2 Frequency of error recognition. The number of 
participants correctly identifying each of the 14 main errors 
built into the simulation. 



We also looked at whether viewing 'high impact' data 
screens impacted the ability of participants to find 
errors. We looked specifically at the two main portal 
pages within our EHR. One was the 'Synopsis' page that 
presents haemodynamics in a graphical format as well as 
all medications and lab values. The other was the 'MD 
Index,' a portal, created by our institution as part of its 
customisation of the EHR, which allows easy access to a 
number of different data screens, including vitals, MAR, 
hemodynamics. We found that use of the Synopsis 
screen was associated with lower performance on the 
simulation. Conversely, use of the MD Index was asso- 
ciated with a significantly better use of the system 
(figure 5). 



80 
70 
60 
50 
40 
30 
20 
10-1 
0 



o 

CD 
i_ 
i— 

o 
o 




g.5 Q. g 3 O O 

V) no 



o 

2 



+- o c 

u u >, 

c c </> 

CO to O 

> > N 




Clinical Medication Failure to 
Change Error Adhere to Best 
practice 



Figure 3 Successful error recognition is mostly independent 
of the training level. Overall recognition rate by fellows (blue) 
and residents (red) for each of the 14 major errors. Data 
analysed by t test. 
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Figure 4 Increased screen utilisation is associated with 
improved performance. The number of independent screens 
visited was correlated with the overall performance on 
simulation. 



DISCUSSION 

In this pilot study, we developed and used a novel 
ICU-specific EHR simulation based on a commonly used 
commercial system. There is an increasing trend to use 
simulation as a tool for assessing end-user competency 
and improving patient safety. A high-fidelity simulation 
allows our study to be conducted in an authentic and 
realistic clinical environment, with the opportunity to 
provide the participant with immediate feedback at the 
conclusion of the simulation. Since end-users often cus- 
tomise their 'user interface' quite significantly, we felt 
that it was important to create a simulation environment 
for our participants that was identical to the actual pro- 
duction EHR environment, including the log-in and key 
clinical screens, and maintain any customisation that 
end-users had already developed. Second, the simulation 
is performed in the ICU on existing clinical workstations 
further enhancing environmental fidelity. Third, the 
case is based on an actual ICU patient and data were 
representative of a typical high-complexity ICU patient 
in terms of the quality, the amount of data within the 
patient chart (including the fact that this was a 5-day 
ICU stay) and the types of errors and safety issues 



A Screen A 
60 



B Screen B 




p=0.01 



Figure 5 Individual screen use correlates with performance. 
The overall success rate was tabulated for user of two of the 
major portals; screens A and B. Overall, use of screen A was 
associated with increased error recognition while screen B 
use was associated with poor performance. Data analysed via 
t test. 
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typically encountered in our ICU. Fourth is our method 
of assessment. By having participants present the patient 
to an ICU physician (either attending or senior fellow), 
we created an environment consistent with our existing 
workflow (as opposed to answering specific questions on 
a written examination, using surveys to elicit informa- 
tion, or recounting the simulation after the passage of 
much time). Finally, the timed nature of the exercise 
was much more consistent with the real workflow in an 
ICU where physicians only have a limited time to search 
for data and was consistent with the existing workflow 
within our ICU. 

Our findings were both surprising and concerning. 
First, only 41.5% of errors were recognised, and while 
fellows performed statistically significandy better than 
interns or residents, their overall performance was still 
below what most would consider acceptable (47%). 
Further, the most severe errors, such as development of 
impending shock, were recognised at even a lower fre- 
quency (40%). We observed the overall poor perform- 
ance among the members of all levels of training, 
despite all of the participants having received general 
training with our EHR and over a years' use with the 
system. Given this finding, it appears that a major stum- 
bling block is the physician interface with the EHR as 
opposed to a pure knowledge deficit. However, these 
observations appear to be in-line with the reported lit- 
erature. Nearly 89% of physicians believe the diagnosis 
of sepsis is missed in the inpatient setting. 31 In patients 
with ARDS, as in this case, nearly 70% of patients are 
still not managed with appropriate ventilator strategies. 34 
Medication errors, including inappropriate dosing due 
to changing renal function, account for nearly 78% of 
the total reported errors in the ICU. Finally, nearly 40% 
of ICU patients are oversedated without acknowledge- 
ment of their sedation score. 35 36 Finally, amongst 
patients who have in-hospital cardiac arrest or need for 
ICU admission, nearly 60% have evidence of clinical 
decompensation prior to transfer and in one study, 
medical staff were only aware of all of the physiologic 
abnormalities in 34% of patients. 37 38 

Our findings are consistent with the description of 
others detailing the ICU as a vulnerable environment 
for the EHR. For example, Han et al documented an 
increased mortality with the introduction of compu- 
terised provider order entry (CPOE) into their Neonatal 
ICU. 21 This was believed not to be due to the system 
itself, but rather due to poor implementation of the 
system, lack of customisation, poor workflow and overall 
poor education and training on how to manage the 
physician-EHR interface. This assessment was supported 
by a subsequent study documenting improved outcomes 
with implementation of an identical system in a similar 
style ICU. 39 40 A similar experience was observed at 
another institution, where an enterprise-wide EHR 
implementation of their EHR proved to be successful, 
with the exception of the MICU. 20 MICU-specific pro- 
blems were attributed to poor training, inadequacies in 



the EHR-physician interface and lack of customisation 
creating unmanageable workflow issues, and the system 
was taken off-line within 6 months. Only after improved 
customisation, increasing the number of available com- 
puters and improved training and education, were they 
able to safely re-introduce the system into their ICU. 20 

While the concept of patient-based simulation in 
general is not new, our study is one of the first to use 
robust, high-fidelity simulation to objectively assess suc- 
cessful use of the EHR and to specifically target identifi- 
cation of changes in the clinical status as the primary 
endpoint. When EHRs have been utilised in simulation 
training, it has often been used with non-physicians such 
as pharmacy or PA students, or rather included in a 
broader simulation exercise where little emphasis was 
placed on the interface with the EHR itself. 27 41 
Interestingly, a recent set of studies from one group has 
used a combination of simulated cases and video analysis 
to assist in EHR design. 42 However, these studies focused 
on CPOE (as opposed to the other functions of EHRs 
including data retrieval) and no data were provided as 
to the fidelity of the simulation or the clinical context of 
the actual cases. 

Within the ICU, two studies have specifically addressed 
the use of EHR simulation. In one, physicians were 
tested about their decision-making with regard to 
end-of-life care in a virtual patient admitted to the ICU 
with metastatic cancer and septic shock. In this scenario, 
the EHR was utilised as a tool for disseminating the case- 
based information while efficient and appropriate use of 
the EHR was not assessed. 29 In the second, researchers 
hypothesised that the user interface to their existing 
EHR decreased efficiency with the system and impaired 
data finding and increased cognitive errors. They had 20 
providers review a case in both their original EHR and 
one with a new front-end to improve data finding, with 
participants answering eight specific questions specific- 
ally related to the management of a bleeding patient. 22 
The new EHR significandy reduced the number of 
incorrect answers to the questions overall, although for 
one question focusing on medications, errors increased. 
This study did have several limitations, including the 
failure to use a high-fidelity environment (use of a 
testing room) , failure to test efficiency with the system 
(no apparent time limit) , a very directed set of questions 
to answer to assess data finding (as opposed to the more 
fluid unknown situation of the average ICU patient) and 
failure to test longitudinal evaluation of data past 24 h. 

The results of our pilot study significantly expand 
upon these prior studies and will allow us to design a 
more robust educational and quality improvement initia- 
tive around EHR simulation. First, we now have a blue- 
print for the creation of additional cases, a prerequisite 
to determine the impact of participation in the simula- 
tion. Second, we have established baseline error recogni- 
tion rates for users at all levels of training and 
experience, thus allowing us to adequately determine 
the sample size required for additional studies. For 
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example, based on the data from cardiac arrest simula- 
tion, we can expect that participation in this exercise 
results in a nearly 20% improvement in error recogni- 
tion on repeat testing, 43 thus requiring at least 10 partici- 
pants at each level of training to undergo repeat testing 
with additional cases to establish this hypothesis. Finally, 
by establishing baseline usability data and simulation 
infrastructure, we now have the ability to also test the 
effect of alterations in the EHR user interface on error 
recognition and overall performance. 

It is important to acknowledge several limitations of 
our study. First, we only tested data retrieval in this part 
of the simulation. We recognise that the EHR affects 
multiple aspects of delivery of care, including communi- 
cation and order entry. However, the process of data 
retrieval, process and recognition is the foundation for 
effective communication and order entry and thus we 
felt a logical place to begin. We plan to expand this 
simulation to address these aspects of the EHR in the 
future. Second the nature and the number of errors 
built in the case. We have discovered, through the 
incorporation of the EHR into our weekly Morbidity and 
Mortality conference, that clinical deterioration in 
patients is often heralded by numerous clinical clues 
and is often caused by a number of small errors within 
an individual case both cognitive and system related. 44 It 
is not uncommon for a patient with nosocomial clinical 
deterioration, as in this case, to have this number of 
issues that need to be identified. However, we also 
acknowledge that care of the average ICU patient 
involves an interprofessional team of pharmacists, nurses 
and respiratory therapists. As a result, until our simula- 
tion is disseminated to all members of the team simul- 
taneously, we cannot be certain that every missed issue 
by the physician will not be caught by other members of 
the team and thus result in direct patient harm. Further, 
it should be stressed that the goal of the simulation is to 
test the system under high-stress/ dangerous situations. 
We believe this is not only a unique aspect in our study, 
but is essential to ensure that the system works optimally 
under all clinical situations. Third, we acknowledge that 
the case created is unique to the ICU environment. 
However, we believe with appropriate case creation, the 
same type of simulation can be used successfully in any 
clinical care environment. Fourth, while the case itself 
was realistic in terms of data presentation and the 
testing was performed in situ, participants were still 
aware that this was a simulated case. As a result, there 
could still exist a significant Hawthorne effect resulting 
in an overestimation of the error recognition rate. 
Finally, the studies were performed utilising one specific 
EHR (EPIC Care). While the most commonly used EHR 
by US physicians, we acknowledge that each EHR and 
user interface will have its own strengths and weaknesses 
in terms of data recognition or processing. 45 However, 
our methods using robust and realistic cases will allow 
other researchers to test the functionality of any other 
EHR. 



In conclusion, implementation of EHRs has brought a 
massive amount of information to the fingertips of ICU 
practitioners across the country. This study demonstrates 
that the combination of sheer data and provider knowl- 
edge is not sufficient for quality patient care: utilisation 
of the EHR is a skill that must be learned. There is 
much room for improvement both in the interface itself 
and how we teach its use. Through the creation of stan- 
dardised cases for EHR simulation, we now have the 
infrastructure to improve user education as well as 
objectively test the efficacy of both new educational tech- 
niques and EHR redesign. 
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